System, information storage medium, and information processing method

ABSTRACT

The system includes a memory that stores a trained model, and a processor. The processor acquires a pre-treatment image in which at least one energy device and at least one biological tissue are imaged, and a state before application of energy is imaged. The processor estimates an estimated heat diffusion region from the pre-treatment image and information regarding an energy supply amount by processing based on a trained model stored in the memory. The processor superimposes the estimated heat diffusion region on a captured image of the camera and displays the captured image with the superimposed estimated heat diffusion region.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Patent ApplicationNo. PCT/JP2022/009692, having an international filing date of Mar. 7,2022, which designated the United States, the entirety of which isincorporated herein by reference. U.S. Patent Applications Nos.63/221,128 and 63/222,252 filed on Jul. 13, 2021 and Jul. 15, 2021 arealso incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

Japanese Unexamined Patent Application Publication No. JP2021-83969discloses a surgical method using an energy device. In this surgicalmethod, an already-ablated biological tissue region and anot-yet-ablated biological tissue region are displayed on a displayusing a computed tomography (CT) image. Then, the next biological tissueregion toward which energy is to be output is presented to the doctor.

SUMMARY OF THE INVENTION

In accordance with one of some aspect, there is provided a systemcomprising:

-   -   a memory configured to store a trained model that is trained to        output a heat diffusion region from a training device tissue        image or a training tissue image, the heat diffusion region        being a range of reach of heat from the at least one energy        device, the training device tissue image being an image in which        at least one energy device that receives energy supply to output        energy and at least one biological tissue are imaged, the        training tissue image being an image in which the at least one        biological tissue is imaged; and    -   a processor,    -   wherein the processor is configured to:    -   acquire a pre-treatment image, in which the at least one energy        device and the at least one biological tissue are imaged, in        which a state before application of energy from the at least one        energy device is imaged, and that is captured by a camera that        captures an image of a surgical field;    -   acquire information regarding an energy supply amount to the at        least one energy device;    -   estimate, based on the pre-treatment image, the information        regarding the energy supply amount, and the trained model, an        estimated heat diffusion region in the pre-treatment image, the        estimated heat diffusion region being an estimated range of        reach of energy from the at least one energy device after        application of the energy based on the energy supply amount; and    -   perform a process of superimposing the estimated heat diffusion        region on a captured image of the camera and displaying the        captured image with the superimposed estimated heat diffusion        region on a display.

In accordance with one of some aspect, there is provided acomputer-readable non-transitory information storage medium storing aprogram for causing a computer to execute

-   -   acquiring a pre-treatment image, in which at least one energy        device and at least one biological tissue are imaged, in which a        state before application of energy from the at least one energy        device is imaged, and that is captured by a camera that captures        an image of a surgical field, and acquiring information        regarding an energy supply amount to the at least one energy        device,    -   estimating an estimated heat diffusion region in the        pre-treatment image by processing based on a trained model, the        estimated heat diffusion region being an estimated range of        reach of energy from the at least one energy device after        application of the energy based on the energy supply amount, the        trained model being trained to output a heat diffusion region        from a training device tissue image or a training tissue image,        the heat diffusion region being a range of reach of heat from        the at least one energy device, the training device tissue image        being an image in which the at least one energy device and the        at least one biological tissue are imaged, the training tissue        image being an image in which the at least one biological tissue        is imaged, and    -   superimposing the estimated heat diffusion region on a captured        image of the camera and displaying the captured image with the        superimposed estimated heat diffusion region on a display.

In accordance with one of some aspect, there is provided an informationprocessing method, comprising:

-   -   acquiring a pre-treatment image, in which at least one energy        device and at least one biological tissue are imaged, in which a        state before application of energy from the at least one energy        device is imaged, and that is captured by a camera that captures        an image of a surgical field, and acquiring information        regarding an energy supply amount to the at least one energy        device,    -   estimating an estimated heat diffusion region in the        pre-treatment image by processing based on a trained model, the        estimated heat diffusion region being an estimated range of        reach of energy from the at least one energy device after        application of the energy based on the energy supply amount, the        trained model being trained to output a heat diffusion region        from a training device tissue image or a training tissue image,        the heat diffusion region being a range of reach of heat from        the at least one energy device, the training device tissue image        being an image in which the at least one energy device and the        at least one biological tissue are imaged, the training tissue        image being an image in which the at least one biological tissue        is imaged, and    -   superimposing the estimated heat diffusion region on a captured        image of the camera and displaying the captured image with the        superimposed estimated heat diffusion region on a display.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a configuration example of a system.

FIG. 2 is a configuration example of a controller.

FIG. 3 is a flowchart for explaining processing performed by acontroller and a system.

FIG. 4 is an example of a pre-treatment image.

FIG. 5 is a configuration example of a monopolar device.

FIG. 6 is a configuration example of a bipolar device.

FIG. 7 is a configuration example of an ultrasonic device.

FIG. 8 is an example of processing according to a first embodiment.

FIG. 9 is a display example of tissue information and deviceinformation.

FIG. 10 is an example of an image labeled with an amount of grippedtissue.

FIG. 11 is an example of an image labeled with an estimated heatdiffusion region.

FIG. 12 is a configuration example of a training device.

FIG. 13 is an explanatory view of a training phase for estimation ofenergy device.

FIG. 14 is an explanatory view of a training phase for estimation oftype of biological tissue.

FIG. 15 is an explanatory view of a training phase for estimation oftension on a treatment target tissue.

FIG. 16 is an explanatory view of a training phase for estimation ofgripping force.

FIG. 17 is an explanatory view of a training phase for estimation ofestimated heat diffusion region.

FIG. 18 is a configuration example of a controller according to a secondembodiment.

FIG. 19 is an example of processing according to the second embodiment.

FIG. 20 is a configuration example of a controller according to a thirdembodiment.

FIG. 21 is a display example of an estimated heat diffusion region inapplication of the third embodiment.

FIG. 22 is an example of processing according to the third embodiment.

FIG. 23 is a display example of an estimated heat diffusion regionshowing a best mode and a worst mode.

FIG. 24 is a table showing the relationship between a combination oftension and gripping force and a corresponding color.

FIG. 25 is a display example of an estimated heat diffusion region inapplication of a fourth embodiment.

FIG. 26 is a configuration example of a controller according to a fifthembodiment.

FIG. 27 is a display example of an estimated heat diffusion region inapplication of the fifth embodiment.

FIG. 28 is a flowchart for explaining processing performed by acontroller and a system in application of the fifth embodiment.

FIG. 29 is a display example of an estimated heat diffusion region whenestimation of gripping amount is not performed.

FIG. 30 is a display example of an estimated heat diffusion region inapplication of a sixth embodiment.

DETAILED DESCRIPTION

The following disclosure provides many different embodiments, orexamples, for implementing different features of the provided subjectmatter. These are, of course, merely examples and are not intended to belimiting. In addition, the disclosure may repeat reference numeralsand/or letters in the various examples. This repetition is for thepurpose of simplicity and clarity and does not in itself dictate arelationship between the various embodiments and/or configurationsdiscussed. Further, when a first element is described as being“connected” or “coupled” to a second element, such description includesembodiments in which the first and second elements are directlyconnected or coupled to each other, and also includes embodiments inwhich the first and second elements are indirectly connected or coupledto each other with one or more other intervening elements in between.

1. System

FIG. 1 is a configuration example of a system 10 according to thepresent embodiment. FIG. 1 shows a configuration example of the systemfor capturing images of a surgical field using an endoscope. The system10 shown in FIG. 1 includes a controller 100, an endoscope system 200, agenerator 300, and an energy device 310. The system 10 is a surgerysystem for performing surgery using at least one energy device under anendoscope. Although an example in which the system 10 includes a singleenergy device 310 is shown, the system 10 may include a plurality ofenergy devices.

The endoscope system 200 is a system that performs imaging by anendoscope, image processing of the endoscope images, and display of theendoscope images in a monitor. The endoscope system 200 includes anendoscope 210, a main body device 220, and a display section 230.Herein, a rigid mirror for surgical operation is described as anexample.

The endoscope 210 includes an insertion section to be inserted into abody cavity, an operation section to be connected to the base end of theinsertion section, a universal cord connected to the base end of theoperation section, and a connector section to be connected to the baseend of the universal cord. The insertion section includes a rigid tube,an objective optical system, an imaging sensor, an illumination opticalsystem, a transmission cable, and a light guide. The objective opticalsystem and the imaging sensor for capturing images inside the bodycavity and the illumination optical system for illuminating the insideof the body cavity are installed in the distal end section of the rigidtube having an elongated cylindrical shape. The distal end section ofthe rigid tube may be configured to be bendable. The transmission cablethat transmits image signals acquired by the imaging sensor, and thelight guide that guides the illumination light to the illuminationoptical system are provided inside the rigid tube. The operation sectionis held by the user and accepts operations from the user. The operationsection has buttons to which various functions are assigned. When thedistal end of the insertion section is bendable, an angle operationlever is provided in the operation section. The connector sectionincludes a video connector that detachably connects the transmissioncable to the main body device 220, and a light guide connector thatdetachably connects the light guide to the main body device 220.

The main body device 220 includes a processing device that controls theendoscope, performs image processing of endoscope images, and displaysthe endoscope images, and a light source device that generates andcontrols illumination light. The main body device 220 is also referredto as a video system center. The processing device is constituted of aprocessor such as a CPU, and performs image processing of the imagesignals transmitted from the endoscope 210 to generate endoscope imagesand then outputs the endoscope images to the display section 230 and thecontroller 100. The illumination light emitted from the light sourcedevice is guided by the light guide to the illumination optical systemand is emitted from the illumination optical system into the bodycavity.

The energy device 310 is a device that outputs energy by high-frequencypower, ultrasonic waves, or the like from its distal end section toperform treatments including coagulation, sealing, hemostasis, incision,division, dissection, or the like, with respect to tissues in contactwith its distal end section. The energy device 310 is also referred toas an energy treatment tool. The energy device 310 may be a monopolardevice in which high-frequency power is energized between an electrodeat the distal end of the device and an electrode outside the body, abipolar device in which high-frequency power is energized between twojaws, an ultrasonic device, which has a probe and a jaw and emitsultrasonic waves from the probe, a combination device in whichhigh-frequency power is energized between the probe and the jaw and alsoemits ultrasonic waves from the probe, or the like.

The generator 300 supplies energy to the energy device 310, controls theenergy supply, and acquires electrical information from the energydevice 310. The generator 300 adjusts output of the energy device 310based on settings made, for example, by a doctor. The generator 300supplies energy corresponding to the settings by the doctor to theenergy device 310, and the energy device 310 receives the energy supplyand performs energy output. When the energy device 310 outputshigh-frequency energy, the generator 300 provides a high-frequencypower, and the energy device 310 outputs the high-frequency power fromthe electrode or jaw. When the energy device 310 outputs ultrasonicenergy, the generator 300 provides electric power, and the probe of theenergy device 310 converts the electric power into ultrasonic waves andoutputs the ultrasonic waves.

The electrical information refers to electrical information of thetissue that comes in contact with the electrode or jaw of the energydevice 310; more specifically, the electrical information is informationobtained as a response to the output of the high-frequency power to thetissue by the energy device 310. The electrical information is, forexample, impedance information of the tissue to be treated by the energydevice 310. However, the electrical information is not limited toimpedance information.

The generator 300 performs control of time-based change in the energyoutput from the energy device 310 according to the output sequence. Thegenerator 300 may vary the energy output according to the time-basedchange in the impedance information. In this case, the output sequencemay specify how the energy output is changed in response to the changein the impedance information. The generator 300 may also automaticallyturn off the energy output according to the time-based change in theimpedance information. For example, the generator 300 may determine thatthe treatment is completed when the impedance rises to a certain levelor higher, and may turn off the energy output.

2. Controller

FIG. 2 is a configuration example of the controller 100. The controller100 includes a control section 110, a storage section 120, I/O devices170, 180, and 190. The controller 100 controls the endoscope system 200,the generator 300, and the like, in this system 10. For example, thecontroller 100 performs various controls through image recognitionprocess using machine learning, and the like. FIGS. 1 and 2 show anexample in which the controller 100 is constituted of a device separatedfrom the generator 300. In this case, the controller 100 is constitutedof an information processing device, such as a PC, a server device, orthe like. Alternatively, the controller 100 may be implemented by, forexample, a system that performs the processes with one or a plurality ofinformation processing devices connected via a network, such as a cloudsystem.

The control section 110 recognizes at least one of the tissueinformation and the treatment information, which is informationregarding the treatment on the biological tissue, from an endoscopeimage through an image recognition process using a trained model 121,and outputs an energy output adjustment instruction based on the imagerecognition information. The energy output adjustment instruction maybe, for example, an instruction based on a surgeon's operation. Thecontrol section 110 includes one or a plurality of processors serving ashardware. The processor is a general-purpose processor such as a CPU(Central Processing Unit), GPU (Graphical Processing Unit), DSP (DigitalSignal Processor), or the like. Alternatively, the processor may be adedicated processor such as an ASIC (Application Specific IntegratedCircuit), an FPGA (Field Programmable Gate Array), or the like.

The storage section 120 stores the trained model 121 used for the imagerecognition process. For example, when the image recognition process isperformed by a general-purpose processor, the storage section 120stores, as the trained model 121, a program that describes an inferencealgorithm and parameters used for the inference algorithm. When theimage recognition process is performed by a dedicated processor with ahardware inference algorithm, the storage section 120 stores theparameters used for the inference algorithm as the trained model 121.The trained model 121 includes a first trained model 122, a secondtrained model 123, a third trained model 124, a fourth trained model125, a fifth trained model 126, and a sixth trained model 127. Eachtrained model is used in each phase of the heat diffusion regionestimation process performed by the present system, as explained in FIG.12 described later. The storage section 120 is a storage device, such asa semiconductor memory, a hard disk drive, an optical disc drive, or thelike. The semiconductor memory is, for example, a RAM, a ROM, anonvolatile memory or the like.

For example, a neural network may be used as the inference algorithm ofthe image recognition process. Weight coefficients and a bias of theinter-node connections in the neural network correspond to theparameters. The neural network includes an input layer to which imagedata is entered, an intermediate layer for performing a calculationprocess with respect to the data input via the input layer, and anoutput layer for outputting recognition results based on the calculationresult output from the intermediate layer. For example, a CNN(Convolutional Neural Network) may be used as the neural network to beused for the image recognition process.

The control section 110 includes a tissue detection section 111, adevice detection section 112, a tissue tension evaluation section 113, agripping force evaluation section 114, a gripping amount evaluationsection 115, a thermal invasion region prediction section 116, and anoutput image creation section 117. The storage section 120 stores aprogram describing the respective functions of the tissue detectionsection 111, the device detection section 112, the tissue tensionevaluation section 113, the gripping force evaluation section 114, thegripping amount evaluation section 115, the thermal invasion regionprediction section 116, and the output image creation section 117. Oneor a plurality of processors of the control section 110 read out theprogram from the storage section 120 and execute the program, therebyrealizing the respective functions of the device detection section 112,the tissue tension evaluation section 113, the gripping force evaluationsection 114, the gripping amount evaluation section 115, the thermalinvasion region prediction section 116, and the output image creationsection 117 of the control section 110. The program describing thefunctions of each of these sections may be stored in a non-transitoryinformation storage medium, which is a computer-readable medium. Theinformation storage medium can be implemented by, for example, anoptical disc, a memory card, an HDD, a semiconductor memory, or thelike. The semiconductor memory is, for example, a ROM or a nonvolatilememory.

The I/O device 180 receives image data of endoscope image from the mainbody device 220 of the endoscope system 200. Further, the I/O device 190sends a signal of the output result of the control section 110 to thedisplay section 230. Each of the I/O devices 180 and 190 is a connectorto which an image transmission cable is connected, or an interfacecircuit connected to the connector to perform communication with themain body device 220.

The I/O device 170 transmits a signal regarding energy output adjustmentinstruction or the like to the generator 300. The energy outputadjustment instruction is, for example, an instruction based on theimage recognition information or a surgeon's operation. Further, the I/Odevice 170 receives a signal related to setting information or the likeof the generator 300. The I/O device 170 is a connector to which asignal transmission cable is connected, or an interface circuitconnected to the connector to perform communication with the generator300.

FIG. 3 is a flowchart for explaining processing performed by thecontroller 100 and the system 10.

First, in the step S1, the control section 110 acquires an endoscopeimage and energy output setting information. The endoscope image can beacquired by the control section 110 from the main body device 220 of theendoscope system 200 via the I/O device 180.

An endoscope image is an image of at least one energy device 310 and atleast one biological tissue and shows a state before application ofenergy from the energy device 310. The endoscope image is an imagecaptured by a camera that captures an image of a surgical field. Thisimage is also referred to as a pre-treatment image. Examples of thecamera that captures an image of a surgical field include, but notlimited to, the endoscope 210. The image shown in FIG. 4 is an exampleof a pre-treatment image. The pre-treatment image shown in FIG. 4 has abipolar device as the energy device 310 and an artery or the like as thebiological tissue. The energy output setting information can be acquiredby the control section 110 from the generator 300 via the I/O device170. The energy output setting information is, for example, energylevel, energy sequence, or similar information. In this way, in the stepS1, the control section 110 acquires information regarding an energysupply amount to be supplied to the energy device 310.

Next, the control section 110 performs the step S2A and the step S2B. Inthe step S2A, the control section 110 detects a tissue in the tissuedetection section 111 based on the pre-treatment image.

In the step S2B, the control section 110 detects jaws 337 and 338 in thedevice detection section 112 based on the pre-treatment image. The jaws337 and 338 are explained in FIG. 6 , which is described later.

Next, the control section 110 performs the step S3A1, the step S3A2, andthe step S3B. In the step S3A1, the control section 110 evaluates thetension applied to a division target tissue in the tissue tensionevaluation section 113 based on the detection results in the step S2Aand the step A2B. The division target tissue refers to a tissue grippedby a doctor with the energy device 310, which is a tissue toward whichenergy is about to output. Further, the tension applied to the divisiontarget tissue refers to the stress exerted on the tissue gripped by theenergy device 310. For example, if a portion of a tissue is gripped andpulled by the energy device 310, the tissue deforms to stretch; in thiscase, the tension applied on the tissue is high. After the step S3A1 isperformed, in the step S3A2, the control section 110 estimates thegripping force by the gripping force evaluation section 114 based on thestep S3A1. The gripping force is the strength of the force in grippingthe division target tissue with the distal end section, e.g., the jaw,of the energy device 310. Further, the control section 110 performs thestep S3B, together with the step S3A1 and the step S3A2.

In the step S3B, the control section 110 estimates the gripping amountbased on the detection results in the step S2A and the step S2B in thegripping amount evaluation section 115. Specifically, the grippingamount is a physical length or area of the portion of the biologicaltissue being gripped by the energy device 310, as explained in FIG. 8described later.

Next, in the step S4, the control section 110 predicts the thermalinvasion region when energy is applied based on information, such asgripped tissue, gripping amount, gripped position, tissue condition,tissue tension, device used, output setting, and the like, in thethermal invasion region prediction section 116. The thermal invasionregion refers to a region where heat diffusion occurs when energy issupplied from the energy device 310, and some changes may be caused inthe biological tissue by the heat. Such change may specifically bethermal damage such as denaturation of proteins, inactivation ofintracellular enzymes, and the like, in biological tissues. In thefollowing, the thermal invasion region will be referred to as a heatdiffusion region, as appropriate. The gripped position is the positionof the portion being gripped by the energy device 310 in the biologicaltissue, which is a treatment target. The gripped position can bepredicted based on the results in the step S2A and the step S2B. Thetissue condition is the state of tissue that can affect the heatdiffusion by the energy device 310. Examples of the tissue conditioninclude the amount of surrounding tissue of the tissue gripped by thejaw, the amount of immersion of the tissue gripped by the jaw or theamount of immersion of surrounding tissue thereof, the amount of fat ofthe tissue gripped by the jaw, and the like. The amount of immersion isan amount of liquid covering the tissue, e.g., an amount of immersion inbody fluids such as blood or lymphatic fluid. The tissue condition canbe predicted, for example, based on the results in the step S2A, etc.described above. The output setting is information regarding the energylevel, energy sequence, and the like, as described above, which isinformation regarding the energy supply amount. The control section 110may acquire this information regarding the energy supply amount, forexample, from the generator 300. The tissue tension is as describedpreviously in the description of the tension of the treatment targettissue, and can be acquired as a result of the step S3A1. The grippedtissue can be acquired from the step S2A, the device used can beacquired from the step S2B, and the gripping amount can be acquired fromthe step S3B.

In this way, in the step S4, the control section 110 estimates theestimated heat diffusion region based on the pre-treatment image, theinformation regarding the energy supply amount, and the trained model121. The estimated heat diffusion region is a region in thepre-treatment image, and is an estimated range of reach of energy afterapplication of the energy from the energy device 310 based on the energysupply amount.

Next, in the step S5, the control section 110 creates an output image inwhich the prediction of the thermal invasion region is superimposed onthe endoscope image in the output image creation section 117.Specifically, the prediction result of the thermal invasion region issuperimposed on the endoscope image by, for example, adding color, to bedisplayed.

Then, finally, in the step S6, the control section 110 displays theresulting image on the display section 230. Specifically, the controlsection 110 outputs the information contained in the image to theendoscope system 200 via the I/O device 190, and the image is displayedon the display section 230 of the endoscope system 200. The displaysection 230 is, for example, a monitor of a personal computer. In thisway, the controller 100 performs a process of causing a display section230 to superimpose the estimated heat diffusion region on a capturedimage of the camera to display the estimated heat diffusion region.

3. Energy Device

In the following, a monopolar device 320, a bipolar device 330, anultrasonic device 340, and a combination device are described asexamples of the energy device 310.

FIG. 5 is a configuration example of the monopolar device 320. Themonopolar device 320 includes an insertion section 322 having anelongated cylindrical shape, an electrode 321 provided at the distal endof the insertion section 322, an operation section 323 connected to thebase end of the insertion section 322, and a cable 325 connecting theoperation section 323 and a connector (not shown). The connector isdetachably connected to the generator 300.

The high-frequency power output by the generator 300 is transmitted bythe cable 325 and output from the electrode 321. A counter electrodeplate is provided outside the patient's body, and energization occursbetween the electrode 321 and the counter electrode plate. This applieshigh-frequency energy to the tissue in contact with the electrode 321,and Joule heat is generated in the tissue. Electrodes having variousshapes are used for the electrode 321 depending on the type of thetreatment. The monopolar device 320 is capable of adjusting the degreeof coagulation and incision by changing the energization pattern.Generally, the target object to be treated by the monopolar device 320is the tissue in contact with the electrode 321, and the heat diffusedaround this tissue in contact with the electrode 321 may affect thesurrounding tissue.

FIG. 6 is a configuration example of a bipolar device 330. The bipolardevice 330 includes an insertion section 332 having an elongatedcylindrical shape, two jaws 337 and 338 provided at the distal endsection 331 of the insertion section 332, an operation section 333connected to the base end of the insertion section 332, and a cable 335connecting the operation section 333 and a connector (not shown). Theconnector is detachably connected to the generator 300. The jaws 337 and338 are movable portions for gripping a tissue and also for applyingenergy to the gripped tissue. The jaws 337 and 338 are structured to beopenable/closable around an axis provided at the base end 336. Theoperation section 333 has a grip section for operating the opening andclosing of the jaws 337 and 338. When the doctor tightly holds the gripsection, the jaws 337 and 338 are closed to grip the tissue.

The high-frequency power output by the generator 300 is transmitted bythe cable 335, and, when the jaws 337 and 338 grip a tissue,energization occurs between the two jaws 337 and 338. As a result,high-frequency energy is applied to the tissue sandwiched between thetwo jaws 337 and 338, Joule heat is generated in the tissue, and thetissue is coagulated. The generator 300 may measure the impedanceinformation of the tissue gripped by the jaws 337 and 338, detectcompletion of the treatment based on the impedance information, and mayautomatically stop the energy output. Further, the generator 300 mayalso automatically adjust the energy applied to the tissue based on theimpedance information. For example, although the device temperature ofthe bipolar device 330 rises only to about 100 degrees Celsius, there isa possibility that a sneak current is generated around the area grippedby the jaws 337 and 338, and heat diffusion may be generated by thesneak current.

A vessel sealing device is a derivative device of the bipolar device.The vessel sealing device is a bipolar device provided with a cutter onits jaw, and separate the tissue by running the cutter after coagulatingthe tissue by energization.

FIG. 7 is a configuration example of the ultrasonic device 340. Theultrasonic device 340 includes an insertion section 342 having anelongated cylindrical shape, a jaw 347 and a probe 348 provided at adistal end section 341 of the insertion section 342, an operationsection 343 connected to the base end of the insertion section 342, anda cable 345 connecting the operation section 343 and a connector (notshown). The connector is detachably connected to the generator 300. Thejaw 347 is movable around an axis provided at the base end 346, and isstructured to be openable/closable with respect to the non-movable probe348. The operation section 343 has a grip section for operating theopening and closing of the jaw 347. When the doctor tightly holds thegrip section, the jaw 347 is closed, and the jaw 347 and the probe 348grip the tissue. The operation section 343 is provided with an operationbutton 344 a to which a first output mode is assigned, and an operationbutton 344 b to which a second output mode is assigned. The output modeis selected according to what treatment is to be performed. When theoperation button for each output mode is pressed, ultrasonic energy isoutput in the output sequence for the corresponding mode.

The power output by the generator 300 is transmitted by the cable 335,and when the operation button 344 a or the operation button 344 b ispressed, the probe 348 converts the power into ultrasonic waves andoutputs the ultrasonic waves. As a result, a frictional heat isgenerated in the tissue sandwiched between the jaw 347 and the probe348, and the tissue is coagulated or incised. Generally, although theheat diffusion of the ultrasonic device 340 is smaller than that of thehigh-frequency device, the device temperature of the ultrasonic device340 can rise to nearly 200 degrees Celsius. The heat diffusion of theultrasonic device 340 is characterized by its tendency to occur in thedirection of the distal end of the probe 348.

The combination device that uses both high-frequency power andultrasonic waves has a configuration similar to that of the ultrasonicdevice shown in FIG. 6 , for example. However, the combination device iscapable of energizing high-frequency power between the jaw and the probeto generate Joule heat in the tissue gripped by the jaw and the probe,thus coagulating the tissue. Similarly to the ultrasonic device, thecombination device is also capable of incising a tissue gripped by thejaw and the probe by outputting ultrasonic waves from the probe. Ahigh-frequency mode is assigned to one of the two operation buttonsprovided on the operation section, and a seal-and-cut mode is assignedto the other one of the two operation buttons. The high-frequency modeis a mode in which coagulation and other treatments are performed usingonly high-frequency energy output. The seal-and-cut mode is a mode inwhich high-frequency energy and ultrasonic energy are used incombination, and the tissue is coagulated and separated byhigh-frequency energy output. With regard to the heat diffusion of thecombination device, for example, heat diffusion similar to either orboth of those of the bipolar device and the ultrasonic device may occur.

In the following embodiment, an exemplary case where the bipolar device330 is mainly used as the energy device 310 is described. However, itshould be noted that the present embodiment is applicable to any casesof using various energy devices mentioned above that may cause heatdiffusion.

4. First Embodiment

FIG. 8 is the first embodiment of an example of processing of thepresent system. First, the control section 110 performs the processingshown in S21A and S21B. Specifically, in the input shown in S21A, thecontrol section 110 acquires a pre-treatment image. Then, in S21B, thecontrol section 110 acquires the device type. The device type can beacquired from the information detected by the device detection section112. Then, in S21A, the control section 110 inputs the acquiredpre-treatment image as output information to the device detectionsection 112, which performs the processing in S22A, and the tissuedetection section 111, which performs the processing in S22B. Further,in S22B, the control section 110 inputs the information regarding theacquired device type as output information to the device detectionsection 112, which performs the processing in S22A.

Next, the control section 110 performs the processing shown in S22A andS22B. In S22A, recognition of the energy device 310 is performed.Specifically, the device detection section 112 detects the energydevices 310 from the pre-treatment image by executing an estimationprogram adjusted by machine learning. The estimation program is aprogram that executes the trained model 121 that has been trained toestimate the type of the energy device 310 from the subject captured inthe pre-treatment image, as explained in FIG. 12 , which is describedlater. In S22A, the device detection section 112 detects the energydevice 310 by inputting the pre-treatment image captured during asurgery acquired in S21A to the network having the estimation program.At this time, the information of the device type acquired in S21B or theinformation regarding the energy supply amount may also be input to thenetwork, together with the pre-treatment image. The targets to beestimated in the processing in S22A also include information such as therange in which the energy device 310 is present, the state of the distalend section of the energy device 310, and the like, in addition to thetype of the energy device 310. The state of the distal end section is,for example, the state whether the jaws 337 and 338 are opened/closed.These types of information about the energy device 310 is referred to asdevice information. The device detection section 112 then labels theportion corresponding to the region of the energy device 310 detected inthe pre-treatment image by, for example, adding a color.

In S22B, the tissue detection section 111 of the control section 110performs recognition of the tissue of the control section 110. Thetissue detection section 111 detects a biological tissue from thepre-treatment image by executing the estimation program. The estimationprogram is a program that executes the trained model 121 that has beentrained to estimate the type of the biological tissue or the like fromthe subject captured in the pre-treatment image, as explained in FIG. 12, which is described later. In S22B, the tissue detection section 111detects the tissue information by inputting the pre-treatment imageacquired in S21A to the network having the estimation program. Thetissue information herein includes the tissue type, the range in whichthe tissue is present, the tissue condition, and the like, of thebiological tissue in the pre-treatment image. The tissue conditionincludes, for example, wet, dry, the presence or absence ofdiscoloration, or the like. Examples of the biological tissue includelarge blood vessels, pancreas, duodenum, as well as the parts connectingthe tissues, vessels such as arteries or veins, and the like. Thebiological tissue may also be simply referred to as a tissue. Then,labeling of the biological tissue type detected above is performed. Thelabeling is performed, for example, by coloring a portion as a region ofa biological tissue. The image shown in FIG. 9 is an example of an imagein which labels are added to the energy device 310 and the treatmenttarget tissue detected in S24A and S24B. In the image shown in FIG. 9 ,each of the energy device 310 and the treatment target tissue islabelled by being surrounded by a frame with textual information. In theexample of FIG. 9 , it is indicated that the treatment target tissue isan artery, and the tissue condition is dry. In this way, the tissuedetection section 111 extracts, from the pre-treatment image, tissueinformation regarding the biological tissue imaged in the pre-treatmentimage.

Then, the pre-treatment image to which labels are added in S22A and S22Bserves as the recognition result in S23. The control section 110 inputsthe recognition result to the tissue tension evaluation section 113,which performs the processing in S24A, and the gripping amountevaluation section 115, which performs the processing in S24B. The inputof the device type in S22A and the input of the tissue information inS22B may be manual input by the doctor. In the tissue recognition inS22B, the detection of biological tissue may also be performed by 3Dmatching with CT (Computed Tomography) or MRI (Magnetic ResonanceImaging), instead of endoscope image.

Next, the control section 110 performs S24A, S24B and S24C. First, inS24A, the gripping amount evaluation section 115 estimates the amount ofgripped tissue, which is the treatment target. Specifically, thegripping amount evaluation section 115 estimates the amount of thegripped tissue, which is the treatment target, based on the recognitionresult in S23 by executing the estimation program. The estimationprogram is a program that executes the trained model 121, which has beentrained to estimate the amount of gripped tissue, which is the treatmenttarget, based on the information regarding the energy device 310 and theinformation regarding the biological tissue. In this program, thegripping amount is determined, for example, according to the size ofoverlap of each segmented subject. In S24A, the gripping amountevaluation section 115 calculates the amount of gripped tissue from therecognized image by inputting the information of the recognition resultin S23 to the network having the estimation program. The tissue grippingamount herein refers to a physical length or area of the portion of thebiological tissue being gripped by the energy device 310. Then, labelingof the tissue gripping amount detected above is performed. The imageshown in FIG. 10 is an example of an image in which a label of thetissue gripping amount estimated in S24A is added by the gripping amountevaluation section 115. As shown in FIG. 10 , the label of the tissuegripping amount is added by displaying a frame surrounding the portiongripping the tissue in the distal end section of the energy device 310.The labeling method is not limited to the example shown in FIG. 10 . Thegripping amount evaluation section 115 then inputs the informationincluding the labelled image to the thermal invasion region predictionsection 116, which performs the processing in S26.

In S24B, the tissue tension evaluation section 113 recognizes thetension condition of the treatment target tissue. Specifically, thetissue tension evaluation section 113 estimates the tension condition ofthe treatment target tissue based on the recognition result in S23 byexecuting the estimation program. As explained in FIG. 12 , which isdescribed later, the estimation program is a program that executes thetrained model 121, which has been trained to estimate the tensioncondition of the treatment target tissue from the information regardingthe energy device 310 and the information regarding the biologicaltissue in S23. If the tension is too weak, the tissue may not be cuteasily and the heat may spread, which may affect the result ofestimation of the heat diffusion region. The tension is as described inFIG. 3 . The tension in the treatment target tissue is also referred toas a tension applied to the biological tissue.

In S24B, the tissue tension evaluation section 113 estimates the tensioncondition of the treatment target tissue from the recognized image byinputting the information of the recognition result in S23 to thenetwork having the estimation program. To estimate the tensioncondition, for example, during the training phase, training is performedby learning how it looks when tension is applied on a certain tissue foreach of various cases, and then, in the actual surgery, estimation isperformed as to how much tension is applied to the tissue currentlygripped by calculating backward from the acquired pre-treatment image.Then, the tissue tension evaluation section 113 inputs the informationabout the estimated tension condition to each of the gripping forceevaluation section 114, which performs the processing in S25, and thethermal invasion region prediction section 116, which performs theprocessing in S26.

In S24C, the control section 110 acquires information about outputsetting and output history. The output setting is the same as the outputsetting information described in FIG. 3 . The output history is historyinformation about energy levels, output settings, and the like, of theoutput from the energy device 310. Examples thereof include informationsuch as the amount of residual heat, or the number of times ofconsecutive cutting before the treatment by the energy device 310. Forexample, immediately after turning off the output of the energy device310, there may be residual heat. Therefore, since the initial conditionsupon heat diffusion are different, the region where heat diffuses isalso different. In this regard, the output history is important in theestimation of the heat diffusion region. These items of information canbe acquired, for example, from the generator 300. The control section110 then inputs the information about the acquired output setting andoutput history to the thermal invasion region prediction section 116.

Next, the control section 110 estimates the gripping force shown in S25.Specifically, the gripping force evaluation section 114 executes theestimation program to estimate the gripping force in gripping the tissueby the energy device 310. The estimation program is a program thatexecutes the trained model 121, which has been trained to estimate thegripping force in gripping the tissue, based on the information oftissue tension condition estimated in S24B. In S25, the gripping forceevaluation section 114 estimates the gripping force by inputting theinformation of tissue tension condition to the network having theestimation program. The method of estimating the gripping force is, forexample, such that, in the training phase, training is performed tolearn the relationship between the gripping force applied by the energydevice 310 and the amount of change in tissue, e.g., the size, color,reflectance, and the like, before and after the gripping regarding thetissue around the grip section. The gripping force is then estimatedfrom the history of the pre-treatment image acquired during the actualsurgery.

The gripping force evaluation section 114 inputs the informationincluding the estimation result to the thermal invasion regionprediction section 116, which performs the processing in S26.

Next, the control section 110 performs prediction of the heat diffusionregion as shown in S26. Specifically, the thermal invasion regionprediction section 116 of the control section 110 executes theestimation program to estimate the range of heat diffusion when theenergy device 310 outputs energy. The estimation program is a programthat executes the trained model 121 that has been trained to estimatethe range of heat diffusion based on information about the type andtension condition of the treatment target tissue, the amount of grippedtissue and force in gripping the tissue by the energy device 310, andthe output setting and output history of the energy device 310. In S26,the thermal invasion region prediction section 116 estimates the heatdiffusion region from the recognized image by inputting such informationto the network having the estimation program. The heat diffusion regionis as described in step S4 in FIG. 3 . Then, labeling of the heatdiffusion region detected above is performed. In this way, the controlsection 110 estimates the estimated heat diffusion region in thetreatment target tissue to be treated by the energy device 310 based onthe pre-treatment image, the information regarding energy supply amount,and the trained model 121. The image shown in FIG. 11 is an example ofan image with a label of estimated heat diffusion region estimated inS26. As shown in FIG. 11 , the heat diffusion regions are shown by beingsurrounded by a frame around the jaw in the distal end section of theenergy device 310. For example, labeling of heat diffusion region on thescreen may be done by coloring the corresponding region. The thermalinvasion region prediction section 116 then inputs the informationincluding the labelled image to the output image creation section 117,which performs the processing in S27.

Finally, the control section 110 performs the output shown in S27.Specifically, the output image creation section 117 of the controlsection 110 creates an image to be displayed on the display section 230of the endoscope system 200 based on the information including the imagecreated by the thermal invasion region prediction section 116 in S26.For example, the output image is created by superimposing theinformation, such as the energy device 310 and the tissue recognized inS22A and S22B, or the gripping amount or the like recognized in S24A, onthe image labelled with the heat diffusion region in S26. In this way,the output image creation section 117 superimposes the estimated heatdiffusion region on the estimated biological tissue region around theenergy device 310 to display the estimated heat diffusion region. Thedisplay of superimposed estimated heat diffusion region may be performedby superimposing the estimated heat diffusion region on the energydevice 310. The output image creation section 117 inputs the outputinformation including the output image thus created to the endoscopesystem 200 via the I/O device 190. Then, the display section 230 of theendoscope system 200 displays the output image, thereby making itpossible to present the heat diffusion region to the doctor.

FIG. 12 is a configuration example of the training device 500, whichperforms machine learning in the present system. The training device 500includes a processing section 510 and a storage section 520. Thetraining device 500 is implemented by an information processing device,such as a PC, a server device, or the like. Alternatively, the trainingdevice 500 may be implemented by a cloud system that performs theprocesses with one or a plurality of information processing devicesconnected via a network.

The processing section 510 is a processor such as a CPU or the like, andthe storage section 520 is a storage device such as a semiconductormemory, a hard disc drive, or the like. The storage section 520 storestraining data 521 and a training model 522. The training data 521includes first training data 521A, second training data 521B, thirdtraining data 521C, fourth training data 521D, fifth training data 521Eand sixth training data 521F. The training model 522 includes a firsttraining model 522A, a second training model 522B, a third trainingmodel 522C, a fourth training model 522D, a fifth training model 522E,and a sixth training model 522F. The processing section 510 uses thetraining data 521 to train the training model 522 to generate a trainedmodel 121.

The training data 521 includes a training device tissue image in whichat least one energy device 310 which receives energy supply and performsenergy output and at least one biological tissue are imaged, or atraining tissue image in which at least one biological tissue is imaged.In each of the training device tissue image and the training tissueimage, correct answer data is added. The correct answer data areannotations in the segmentation (region detection) in machine learning,annotations in the detection (location detection), correct answer labelsin the classification (classification), or correct answer labels in theregression (regression analysis). In the following description, thetraining device tissue image and the training tissue image may becollectively referred to as a training image.

The first training data 521A is the training data 521 regarding theenergy device 310. The second training data 521B, the third trainingdata 521C, the fourth training data 521D, the fifth training data 521E,and the sixth training data 521F are training data regarding thebiological tissue, the amount of gripped biological tissue, the tensioncondition of the biological tissue, the gripping force in gripping thebiological tissue, and the heat diffusion range, respectively. Thetraining model 522 has the same correspondence, i.e., the first trainingmodel 522A, the second training model 522B, the third training model522C, the third training model 522D, the third training model 522E, andthe third training model 522F are training model regarding the energydevice 310, the biological tissue, the amount of gripped biologicaltissue, the tension condition of the biological tissue, the grippingforce in gripping the biological tissue, and the heat diffusion range,respectively. For example, the processing section 510 inputs a trainingimage, which is the first training data 521A about the energy device310, to the inference process by the first training model 522A about theenergy device 310. Then, feedback is given to the first training model522A based on the error between the results of the inference process andthe first training data 521A. This process is repeated using a largenumber of first training data 521A, thereby the first trained model 122can be generated. In this way, it becomes possible to realize theestimation of the energy device 310 at a higher accuracy in a variety ofsurgical situations. The same can be said for each of the other trainingdata, training models, and trained models. The processing section 510then transfers the trained model 121 thus generated to the controller100, and the trained model 121 is stored in the storage section 120 ofthe controller 100.

FIGS. 13 to 17 explain the details of the aforementioned training phase.FIG. 13 is an explanatory view of the first trained model 122 used forthe estimation of the energy device 310 in the device detection section112. As shown in FIG. 13 , in the training device 500, the firsttraining data 521A labeled with annotations corresponding to thepre-treatment image of the energy device 310 and the biological tissueis fed back to the first training model 522A to modify the existingfirst trained model 122, and the new first trained model 122 is input tothe controller 100. The content of the annotation is correct answerdata, such as the type, the position, and the range of the presence ofthe energy device 310, or the configuration and the condition or thelike of the distal end section of the energy device 310.

FIG. 14 is an explanatory view of the second trained model 123 used forthe estimation of biological tissue in the tissue detection section 111.Similarly to the case shown in FIG. 13 , in the training device 500, thesecond training data 521B labeled with annotations corresponding to thepre-treatment image is fed back to the second training model 522B tomodify the existing second trained model 123, and the new second trainedmodel 123 is input to the controller 100. The content of the annotationis correct answer data, such as the name of the tissue present in thepre-treatment image, the range of the presence of each tissue, thecondition of each tissue, and the like.

FIG. 15 is an explanatory view of the fourth trained model 125 used forthe estimation of the tension applied on the treatment target tissue inthe tissue tension evaluation section 113. In the training device 500,the fourth training data 521D labeled with annotations corresponding tothe pre-treatment image is fed back to the fourth training model 522D tomodify the existing fourth trained model 125, and the new fourth trainedmodel 125 is input to the controller 100. The content of the annotationis correct answer data, such as the name of the treatment target tissueor the range of the presence of the treatment target tissue, the amountof tension applied in the region, and the like.

The fourth training data 521D for the tension application amount can beacquired, for example, from the setting of the energy device 310.

FIG. 16 is an explanatory view of the fifth trained model 126 used forthe estimation of gripping force in the gripping force evaluationsection 114. In the training device 500, the fifth training data 521Elabeled with annotations corresponding to the pre-treatment image is fedback to the fifth training model 522E to modify the existing fifthtrained model 126, and the new fifth trained model 126 is input to thecontroller 100. The contents of the annotations are correct answer data,including the tension application amount, the amount of change of tissuearound the grip section, the gripping force, and the like. The fifthtraining data 521E for the tension application amount can be acquired,for example, from the setting of the energy device 310, similarly to thecase of FIG. 15 . The gripping force can also be acquired from thesetting of the energy device 310. The amount of change of tissue aroundthe grip section can be extracted, for example, from the historyinformation of the pre-treatment image.

FIG. 17 is an explanatory view of the sixth trained model 127 used forthe estimation of the heat diffusion region in the thermal invasionregion prediction section 116. In the training device 500, the sixthtraining data 521F labeled with annotations corresponding to thepre-treatment image is fed back to the sixth training model 522F tomodify the existing sixth trained model 127, and the new sixth trainedmodel 127 is input to the controller 100. The contents of theannotations are correct answer data, including the tissue type, thetissue condition, the gripping amount, the application amount of tissuetension, the gripping force, the output setting, the type of the energydevice 310, the output history, and heat diffusion region after thetreatment, and the like. For example, the doctor may perform labeling ofsome of the correct answer data.

One of the keys to usual energy treatment in surgery is to suppress heatdiffusion from the energy device to avoid thermal damages to surroundingorgans. However, because the tissues to be treated are not uniform, thetime required for the procedure, such as division, varies due to thedifference in tissue type, the difference in tissue condition,individual differences of the patients, or the like; accordingly, thedegree of heat diffusion also varies. To cope with these issues andsuppress the heat diffusion, the doctors have been adjusting the amountof the tissue gripped by the energy device and the tissue tension;however, an appropriate adjustment may be difficult in some cases, inparticular for non-experts with fewer experiences. Therefore, in orderto proceed with the manipulation more efficiently, it is desirable tohave support from the system.

As described above, in the treatments using energy devices, it is oftennecessary to watch the heat diffusion to the surrounding area, and thedoctors perform the treatments while estimating the degree of heatdiffusion. In the Japanese Unexamined Patent Application Publication No.2021-83969 described above, an already-ablated biological tissue regionand a not-yet-ablated biological tissue region are displayed on adisplay to indicate to the doctor the region of biological tissue towhich energy should be output next. However, because the output isperformed by estimating the temperature change from the differencebetween the CT image before the energy output and the CT image after thestart of energy output, only the temperature change at or after thestart of heat output can be estimated, and the appropriate position ofthe treatment tool cannot be presented before the start of heat output.In addition, the range of a critical tissue is unknown in some cases.

In this regard, according to the present system, the estimation of theheat diffusion region is performed based on the information of theenergy device, the biological tissue, and the like, and the estimatedheat diffusion region is superimposed on the display screen. This allowsthe doctor to grasp in advance the heat diffusion region and make theoutput setting of the energy device in a way such that thermal damagesto the treatment target tissue can be avoided. In addition, the presentsystem performs the estimation of the heat diffusion region by usingmachine learning, thus making it possible to perform safe and efficientsurgery and improve stability in surgery regardless of the doctor'sexperience.

5. Second Embodiment

FIG. 18 is a configuration example of the controller 100 according tothe second embodiment of the present system. The gripping forceevaluation section 114 of the second embodiment differs from that in thefirst embodiment shown in FIG. 2 . Specifically, in the firstembodiment, the information input to the gripping force evaluationsection 114 is output from the tissue tension evaluation section 113 andthe gripping amount evaluation section 115; however, in the secondembodiment, the information input to the gripping force evaluationsection 114 is output from the generator 300 provided outside thecontroller 100. That is, in the second embodiment, the gripping forceevaluation section 114 detects the gripping force of the energy device310 by acquiring information from the generator 300 provided outside thecontroller 100. The generator 300 is capable of acquiring a detectionvalue of the gripping force from, for example, a gripping forcedetection sensor, such as a stress sensor, a position meter, or thelike, mounted on the handle of energy device 310. Then, the controlsection 110 estimates the estimated heat diffusion region based on thepre-treatment image, the information regarding the energy supply amount,and the gripping force acquired from the gripping force detectionsensor.

FIG. 19 is an example of the processing according to the secondembodiment of the present system. Compared to the aforementioned exampleof processing in the first embodiment in FIG. 8 , in FIG. 19 , themeasurement of gripping force shown in S34C is performed instead of theestimation of gripping force in S25. Further, in the second embodiment,the gripping force evaluation section 114 does not acquire the outputresult of the tissue tension evaluation section 113, but acquires thedetection value of the gripping amount from an external device, such asthe generator 300.

According to the second embodiment, by using the data measured by thegripping force detection sensor of the energy device 310, the estimationprocess in the gripping force evaluation section 114 can be skipped.This enables acceleration of the process of estimation of the heatdiffusion region. In addition, in some cases, the certainty factor ofthe base data may be low in the estimation process in the gripping forceevaluation section 114. In this case, even if a certain period of timeis taken for the estimation process, the certainty factor of theestimation result may also be low, and if the doctor performs a surgeryusing such uncertain estimation results, it will be difficult to performan efficient surgery. Therefore, according to the second embodiment,efficient surgery can be performed while maintaining more safety.

6. Third Embodiment

FIG. 20 is a configuration example of the controller 100 according tothe third embodiment of the present system. The third embodiment differsfrom the second embodiment shown in FIG. 18 in that the tissue tensionevaluation section 113 is not provided. That is, in the thirdembodiment, the tissue tension is not estimated by machine learning;instead, the estimation of the heat diffusion region is performed. Then,the heat diffusion region that varies depending on the applicationamount of tissue tension, i.e., the tension applied to the biologicaltissue, is superimposed on the display screen. For example, both theheat diffusion prediction range when the tension is intense and the heatdiffusion prediction range when the tension is low are presented.Further, it is also possible to estimate an estimated heat diffusionregion corresponding to each of a plurality of stages of tensionintensity. In this case, the plurality of stages may be gradations. FIG.21 shows application of the third embodiment, showing an example of animage in which the estimated heat diffusion region when the tension isintense and the estimated heat diffusion region when the tension is weakare respectively superimposed on the pre-treatment image to bedisplayed. When the tension applied on the tissue is intense, theestimated heat diffusion region is displayed in a narrow range on bothsides of the jaw at the distal end section of the energy device 310.Then, when the tension applied on the tissue is weak, the estimated heatdiffusion region is displayed in a wide range on both sides of the jaw.

FIG. 22 shows an example of the processing according to the thirdembodiment. Compared to the aforementioned example of the processing inthe first embodiment in FIG. 3 , the step of evaluating the tensionapplied to the tissue in the step S3A1 is omitted. There is also no stepof estimating the gripping force in the gripping force evaluationsection 114 shown in S3A2 of FIG. 3 . Further, in the third embodimentshown in FIG. 22 , in the step S45, the prediction of the thermalinvasion region is displayed by being superimposed on the endoscopeimage for each amount of tension applied to the tissue.

In this way, in the third embodiment, the estimation process in thetissue tension evaluation section 113 and the gripping force evaluationsection 114 can be skipped. Therefore, the process of estimating theheat diffusion region can be further accelerated compared to the case ofthe second embodiment.

Further, as a modification example of the third embodiment, the controlsection 110 may estimate the estimated heat diffusion regioncorresponding to each of the plural stages of the gripping amountwithout estimating the gripping amount of the energy device 310 imagedin the pre-treatment image. In this way, the estimation process in thegripping amount evaluation section 115 can be skipped, thus the processof estimating the estimated heat diffusion region can be accelerated.

7. Fourth Embodiment

As in the first and second embodiments, after measuring the tensionapplied to the tissue and the amount of tissue gripped by the energydevice 310, if the certainty factors of the estimated tension and thegripping amount are lower than a predetermined value, the best case andthe worst case for the tension and the gripping amount may be displayed.Further, in this case, it is also possible to estimate the estimatedheat diffusion region corresponding to each of a plurality of stages foreach parameter. That is, in the fourth embodiment, for example, if thecertainty factor in the estimation of the tension applied to the tissueis lower than the first predetermined value, the estimated heatdiffusion region corresponding to each of plural stages of tension isestimated without using the estimated tension. Then, if the certaintyfactor in the estimation of the gripping amount is lower than the secondpredetermined value, the heat diffusion region corresponding to each ofplural stages of gripping force is estimated without using the estimatedgripping force. The first predetermined value and the secondpredetermined value are reference values for determining that thecertainty factor of the tension value estimated by machine learning islow and therefore is inappropriate in use for the estimation of theestimated heat diffusion region, and can be set arbitrarily by thedoctor, for example. In the present embodiment, the doctor will be ableto select appropriate energy setting from the displayed information ofthe estimated heat diffusion region for each stage, thereby can performthe surgery safely and efficiently. Further, in the fourth embodiment,if the certainty factors of other parameters than the tension orgripping amount are lower than the predetermined value, the estimatedvalue of the parameter may not be used to estimate the estimated heatdiffusion region. The fourth embodiment may be applied when thecertainty factor of either or both of the tension and the grippingamount is low. The first predetermined value and the secondpredetermined value may be the same or different.

Further, in the present embodiment, the control section 110 may alsoestimate the estimated heat diffusion region for each temperature orheat, and superimpose the estimated heat diffusion region for eachtemperature or heat thus estimated on the pre-treatment image to displaythe estimated heat diffusion region. That is, the control section 110may superimpose the estimated heat diffusion region around therecognized energy device 310 and the biological tissue by changing thecolor for each variable, temperature, and heat to display the estimatedheat diffusion region.

Regarding the color, for example, a color tone between the color fordisplaying the tension and the color for displaying the gripping forcemay be used for the display. For example, as shown on the left side ofFIG. 23 , the tension has a warm color, for example, red is shown whenthe tension is intense and orange is shown when the tension is weak, andthe gripping force has a cold color, for example, blue is shown when thegripping force is intense and dark blue is shown when the gripping forceis weak. In this case, as shown on the right side of the same figure,upon the display of the estimated heat diffusion region, they areintegrated and displayed with intermediate colors. Then, for theparameters of the tension and the gripping force, display indicatingthat the accuracy is insufficient may be performed. Further, forexample, it is possible to display information indicating that the bestmode is a case where the tension and the gripping amount are intense,which makes the estimated heat diffusion region narrowest, and that theworst mode is a case where the tension and the gripping force are weak,which makes the estimated heat diffusion region largest. In this way,the doctor will be able to grasp which parameters have a low certaintyfactor and will be able to determine, based on the color, whatcombination of the plural parameters with low certainty factors can beused to set the best mode. It is not easy for a doctor to determine whattype of combination of parameter setting would minimize the heatdiffusion region when there are a plurality of parameters having a lowcertainty factor. Therefore, in this way, it is possible to support thedoctor's decision by presenting a combination that minimizes the heatdiffusion region from among the possible combinations of the parametersby using machine learning.

Further, FIGS. 24 and 25 are examples in which, when there are aplurality of parameters having a low certainty factor, a color is setfor each of all possible combinations of the setting values of theparameters, and is displayed on the screen. FIG. 24 shows all possiblecombinations of the tension and the gripping force with intense or weaksetting values, and the colors corresponding to these combinations. FIG.25 is an example of an image showing how the estimated heat diffusionregions are distributed by showing the corresponding colors, when theparameter combinations shown in FIG. 24 are used. In the case shown inFIG. 25 , it can be seen that the estimated heat diffusion regionbecomes narrowest when both the tension and the gripping force areintense.

8. Fifth Embodiment

FIG. 26 is a configuration example of the controller 100 according tothe fifth embodiment of the present system. The fifth embodiment differsfrom the first embodiment shown in FIG. 2 in that the gripping forceevaluation section 114 is omitted. That is, in the fifth embodiment, theestimation of the heat diffusion region is performed without estimatingthe gripping force by machine learning. Then, the heat diffusion regionthat varies depending on the gripping force is superimposed on thedisplay screen. At this time, the information of gripping force is inputto the fifth training data 521E, and the estimated heat diffusionregions for various degrees of gripping force are estimated andsuperimposed on the pre-treatment image and are displayed. FIG. 27 showsapplication of the fifth embodiment, showing an example of an image inwhich the estimated heat diffusion region when the gripping force isintense and the estimated heat diffusion region when the gripping forceis weak are respectively superimposed on the pre-treatment image and aredisplayed. In the case shown in FIG. 27 , when the gripping force of theenergy device 310 is intense, the estimated heat diffusion region isdisplayed in a narrow range on both sides of the jaw at the distal endsection of the energy device 310. Then, when the gripping force is weak,the estimated heat diffusion region is displayed in a wide range on bothsides of the jaw. Further, in this case, it is also possible to estimatethe estimated heat diffusion region corresponding to each of a pluralityof stages of the degree of gripping force.

FIG. 28 is a flowchart for explaining processing performed by thecontroller 100 and the system 10 in application of the fifth embodiment.Compared to the flowchart shown in FIG. 3 , the step of estimating thegripping force in the gripping force evaluation section 114 in the stepS3A2 of FIG. 3 is omitted. Further, in the step S5 of FIG. 3 , in thefifth embodiment, the prediction of the thermal invasion region issuperimposed on the endoscope image for each gripping force applied tothe tissue to create the output image.

According to the fifth embodiment, the heat diffusion region that variesdepending on the gripping force can be superimposed on the displayscreen to be displayed without measuring the gripping force. Therefore,it is not necessary to provide a sensor or the like for detecting thegripping force or the like in the energy device 310 or the like, therebythe present system can be realized at a low cost. Further, sterilizationof the energy device 310 may also be simplified.

Although the fifth embodiment is a case where the estimated heatdiffusion region is estimated without estimating the gripping force, thesame can be applied to other parameters, for example, the grippingamount of the biological tissue. FIG. 29 is an example of an image inwhich the estimated heat diffusion regions are superimposed in differentpatterns respectively for a case with a deep gripping amount and a casewith a shallow gripping amount, without performing the estimation of thegripping amount.

9. Sixth Embodiment

FIG. 30 is a display example of estimated heat diffusion region inapplication of the sixth embodiment of the present system. As shown inFIG. 30 , the estimated heat diffusion regions at 0.1 second, 5 seconds,and 10 seconds after the start of energy application are respectivelysuperimposed on the endoscope image and are displayed. For example, evenif the estimated heat diffusion region immediately after the output isnarrow, the estimated heat diffusion region may soon become widerthereafter depending on the heat conduction and the tissue condition ofthe treatment target tissue and surrounding tissue thereof. Therefore,according to the sixth embodiment, the doctor can grasp the extent towhich the estimated heat diffusion region extends as a result ofcontinuous application of energy, thereby can perform the surgery moresafely and efficiently.

Further, the system of the present embodiment can also be realized as aprogram. Specifically, a pre-treatment image, in which at least oneenergy device and at least one biological tissue are imaged, in which astate before application of energy from the energy device is imaged, andthat is captured by a camera that captures an image of a surgical field,is acquired, and information regarding the energy supply amount to besupplied to the energy device is acquired. Further, an estimated heatdiffusion region, which is an estimated range of reach of the energyafter the application of energy from the energy device based on theenergy supply amount in the pre-treatment image is estimated byprocessing based on a trained model that has been trained to output,from the training device tissue image, in which at least one energydevice and at least one biological tissue are imaged, or from thetraining tissue image, in which the at least one biological tissue isimaged, a heat diffusion region that is a range of reach of heat fromthe energy device. Then, a computer is caused to perform a process ofcausing a display section to superimpose the estimated heat diffusionregion on a captured image of the camera to display the estimated heatdiffusion region. The computer assumed herein may be a network terminalor the like, such as a personal computer or the like. However, thecomputer may also be a wearable terminal such as a smartphone, a tablet,a smartwatch, or the like. In this way, the same effects as thosedescribed above can be achieved.

Further, the system of the present embodiment can also be realized as aninformation processing method. Specifically, in the informationprocessing method, a pre-treatment image, in which at least one energydevice and at least one biological tissue are imaged, in which a statebefore application of energy from the energy device is imaged, and thatis captured by a camera that captures an image of a surgical field, isacquired, and information regarding the energy supply amount to besupplied to the energy device is acquired. Further, an estimated heatdiffusion region, which is an estimated range of reach of the energyafter the application of energy from the energy device based on theenergy supply amount in the pre-treatment image is estimated byprocessing based on a trained model that has been trained to output,from the training device tissue image, in which at least one energydevice and at least one biological tissue are imaged, or from thetraining tissue image, in which the at least one biological tissue isimaged, a heat diffusion region that is a range of reach of heat fromthe energy device. Then, the estimated heat diffusion region issuperimposed on a captured image of the camera, and is displayed on thedisplay section. In this way, the same effects as those described abovecan be achieved.

The system 10 of the present embodiment described above includes thestorage section 120 that stores the trained model 121 and the controlsection 110. The trained model 121 is trained to output a heat diffusionregion from the training device tissue image or the training tissueimage. The training device tissue image is an image in which at leastone energy device, which receives energy supply and performs energyoutput, and at least one biological tissue are imaged. The trainingtissue image is an image in which at least one biological tissue isimaged. The control section 110 acquires a pre-treatment image, in whichat least one energy device and at least one biological tissue areimaged, in which a state before application of energy from the energydevice 310 is imaged, and that is captured by a camera that captures animage of a surgical field. The control section 110 acquires informationregarding the energy supply amount supplied to the energy device. Thecontrol section 110 estimates, based on the pre-treatment image, theinformation regarding the energy supply amount, and the trained model,an estimated heat diffusion region, which is an estimated range of reachof energy from the energy device after application of the energy basedon the energy supply amount, in the pre-treatment image. Then, thecontrol section 110 causes a display section to superimpose theestimated heat diffusion region on a captured image of the camera todisplay the estimated heat diffusion region.

As a result, in some embodiments, the heat diffusion region is estimatedbased on the pre-treatment image and the information regarding theenergy supply amount, and the estimated heat diffusion region issuperimposed on the display screen and is displayed. This allows thedoctor to grasp in advance the heat diffusion region and make the outputsetting of the energy device 310 in a way such that thermal damages tothe treatment target tissue can be avoided. In addition, by performingthe estimation of the heat diffusion region using machine learning, itis possible to perform safe and efficient surgery and improve stabilityin surgery regardless of the doctor's experience. The tissue informationis described, for example, in the section “1. System”.

Further, in the present embodiment, the control section 110 may alsoextract the tissue information, which is information regarding thebiological tissue imaged in the pre-treatment image, from thepre-treatment image, and estimate the estimated heat diffusion regionbased on the tissue information and the information regarding the energysupply amount.

As a result, in some embodiments, the treatment target tissue can beextracted by machine learning using pre-treatment images. Therefore, theestimated heat diffusion region can be estimated based on this.

Further, in the present embodiment, the control section 110 may alsoextract, from the pre-treatment image, device information, which isinformation regarding the energy device 310 imaged in the pre-treatmentimage, and may estimate the estimated heat diffusion region in thetreatment target tissue to be treated by the energy device 310 based onthe device information and the tissue information.

As a result, in some embodiments, the treatment target tissue and theenergy device 310 can be extracted by machine learning usingpre-treatment images. Therefore, the estimation of estimated heatdiffusion region can be performed based on these items of information.The device information is described in the section “4. FirstEmbodiment”.

Further, in the present embodiment, the control section 110 estimatesregions of the treatment target tissue and the energy device 310 in thepre-treatment image, based on the tissue information and the deviceinformation. The control section 110 estimates, based on the estimatedregions of the treatment target tissue and the energy device 310, atleast one of the tension applied to the biological tissue imaged in thepre-treatment image and the gripping amount of the energy device as anestimation result. The control section 110 estimates the estimated heatdiffusion region based on the estimation result, the tissue information,and the information regarding the energy supply amount.

As a result, in some embodiments, the estimated heat diffusion regioncan be estimated by machine learning using the estimation results forthe tension applied to the biological tissue and the gripping amount ofthe energy device. The tension applied to the biological tissue isdescribed in the section “2. Controller” and the gripping amount of theenergy device is described in the section “4. First Embodiment”.

Further, in the present embodiment, the control section 110 may estimatethe gripping force of the energy device 310 based on the estimatedtension and the pre-treatment image, and may estimate the estimated heatdiffusion region based on the estimated gripping force.

As a result, in some embodiments, the gripping force of the energydevice 310 can be estimated by machine learning based on the tensionestimated by the control section 110 and the pre-treatment image.Therefore, it is possible to estimate the estimated heat diffusionregion using the gripping force.

Further, in the present embodiment, the control section 110 may alsoacquire a detection value of the gripping force detected by a grippingforce detection sensor provided in the grip section of the energy device310, and may estimate the estimated heat diffusion region based on theacquired detection value.

As a result, in some embodiments, by using the data measured by thegripping force detection sensor, the estimation process in the grippingforce evaluation section 114 can be skipped. This enables accelerationof the process of estimation of the estimated heat diffusion region. Inaddition, when the certainty factor of the data of the gripping force,which is used as the base of the estimation, is low, the data measuredby the gripping force detection sensor can be used to achieve safe andefficient surgery.

Further, in the present embodiment, the control section 110 may alsoestimate the estimated heat diffusion region for each temperature orheat, and superimpose the estimated heat diffusion region for eachtemperature or heat thus estimated on the pre-treatment image to displaythe estimated heat diffusion region.

As a result, in some embodiments, the control section 110 is capable ofsuperimposing the estimated heat diffusion region around the recognizedenergy device 310 and the biological tissue by changing the color foreach variable, temperature, and heat, to display the estimated heatdiffusion region.

Further, in the present embodiment, the control section 110 may alsoestimate the region of the biological tissue and the region of theenergy device 310 from the pre-treatment image, and superimpose theestimated heat diffusion region on the estimated region of biologicaltissue around the energy device 310 to display the estimated heatdiffusion region.

As a result, in some embodiments, the region of the biological tissueand the region of the energy device 310 can be estimated from thepre-treatment image by machine learning. Therefore, it is possible todisplay the estimated heat diffusion region by superimposing it on thepre-treatment image.

Further, in the present embodiment, the control section 110 may estimatethe estimated heat diffusion region corresponding to each of pluralstages of the tension without estimating the tension applied to thebiological tissue imaged in the pre-treatment image.

As a result, in some embodiments, the estimation process in the tissuetension evaluation section 113 can be skipped. Therefore, the process ofestimating the estimated heat diffusion region can be accelerated. Themethod of displaying the estimated heat diffusion region correspondingto each of the plural stages of the tension is described in FIG. 21 inthe section “6. Third Embodiment”.

Further, in the present embodiment, the control section 110 may estimatethe estimated heat diffusion region corresponding to each of the pluralstages of the gripping amount without estimating the gripping amount ofthe energy device 310 imaged in the pre-treatment image.

As a result, in some embodiments, the estimation process in the grippingamount evaluation section 115 can be skipped, thus the process ofestimating the estimated heat diffusion region can be accelerated. Themethod of displaying the estimated heat diffusion region correspondingto each of the plural stages of the gripping amount is described in FIG.29 in the section “8. Fifth Embodiment”.

Further, in the present embodiment, the control section 110 estimates,based on the pre-treatment image, the tension applied to the biologicaltissue imaged in pre-treatment image and the gripping amount of theenergy device 310. If the certainty factor in the estimation of thetension is lower than the first predetermined value, the control section110 estimates the estimated heat diffusion region corresponding to eachof the plural stages of the tension without using the estimated tensionfor the estimation of the estimated heat diffusion region. If thecertainty factor in the estimation of the gripping amount is lower thanthe second predetermined value, the control section 110 estimates theestimated heat diffusion region corresponding to each of the pluralstages of gripping amount without using the estimated gripping force forthe estimation of the estimated heat diffusion region.

As a result, in some embodiments, it is possible to determine whether ornot to use estimation values of the tension and the gripping amount forthe estimation of the heat diffusion region depending on the certaintyfactors of these estimation values. In this way, when the certaintyfactor of the estimation value is low, it is possible to estimate theheat diffusion region corresponding to each of the plural stages of thetension and the gripping amount without using the estimation value.Therefore, the doctor will be able to select appropriate energy settingfrom the displayed information, thereby performing the surgery safelyand efficiently. The first predetermined value and the secondpredetermined value are described in the section “7. Fourth Embodiment”.The method of displaying the estimated heat diffusion regioncorresponding to each of the plural stages of the tension and thegripping amount is described in FIG. 23 in the section “7. FourthEmbodiment”.

Further, in the present embodiment, the control section 110 may estimatethe estimated heat diffusion region corresponding to each of the pluralstages of the force of gripping the biological tissue by the energydevice 310.

As a result, in some embodiments, the heat diffusion region that variesdepending on the gripping force can be displayed while beingsuperimposed on the display screen without measuring the gripping force.Therefore, it is not necessary to provide a sensor or the like fordetecting the gripping force or the like in the energy device 310 or thelike, thereby the present system can be realized at a low cost. Anexample of the display of the estimated heat diffusion region using thepresent embodiment is shown in FIG. 27 .

Further, in the present embodiment, the control section 110 may estimatethe estimated heat diffusion region corresponding to each of the pluralstages of time for the energy device 310 to apply energy.

As a result, in some embodiments, the doctor can grasp the extent towhich the estimated heat diffusion region extends as a result ofcontinuous application of energy, thereby can perform the surgery moresafely and efficiently. The method of displaying the estimated heatdiffusion region corresponding to each of the plural stages of time isdescribed in FIG. 30 in the section “9. Sixth Embodiment”.

Further, the above processing may also be written as a program. That is,the program of the present embodiment causes the controller 100 toexecute acquiring a pre-treatment image, acquiring information regardingan energy supply amount, estimating an estimated heat diffusion regionthat is an estimated range of reach of energy by processing based on thetrained model 121, and superimposing the estimated heat diffusion regionon a captured image of a camera to display the estimated heat diffusionregion in a display section.

Further, the above processing may also be written as an informationprocessing method. That is, the information processing method of thepresent embodiment acquires a pre-treatment image, acquires informationregarding an energy supply amount, estimates an estimated heat diffusionregion that is an estimated range of reach of energy by processing basedon the trained model 121, and superimposes the estimated heat diffusionregion on a captured image of a camera, to display the estimated heatdiffusion region in a display section.

Although the embodiments to which the present disclosure is applied andthe modifications thereof have been described in detail above, thepresent disclosure is not limited to the embodiments and themodifications thereof, and various modifications and variations incomponents may be made in implementation without departing from thespirit and scope of the present disclosure. The plurality of elementsdisclosed in the embodiments and the modifications described above maybe combined as appropriate to implement the present disclosure invarious ways. For example, some of all the elements described in theembodiments and the modifications may be deleted. Furthermore, elementsin different embodiments and modifications may be combined asappropriate. Thus, various modifications and applications can be madewithout departing from the spirit and scope of the present disclosure.Any term cited with a different term having a broader meaning or thesame meaning at least once in the specification and the drawings can bereplaced by the different term in any place in the specification and thedrawings.

1. A system comprising: a memory configured to store a trained model that is trained to output a heat diffusion region from a training device tissue image or a training tissue image, the heat diffusion region being a range of reach of heat from the at least one energy device, the training device tissue image being an image in which at least one energy device that receives energy supply to output energy and at least one biological tissue are imaged, the training tissue image being an image in which the at least one biological tissue is imaged; and a processor, wherein the processor is configured to: acquire a pre-treatment image, in which the at least one energy device and the at least one biological tissue are imaged, in which a state before application of energy from the at least one energy device is imaged, and that is captured by a camera that captures an image of a surgical field; acquire information regarding an energy supply amount to the at least one energy device; estimate, based on the pre-treatment image, the information regarding the energy supply amount, and the trained model, an estimated heat diffusion region in the pre-treatment image, the estimated heat diffusion region being an estimated range of reach of energy from the at least one energy device after application of the energy based on the energy supply amount; and perform a process of superimposing the estimated heat diffusion region on a captured image of the camera and displaying the captured image with the superimposed estimated heat diffusion region on a display.
 2. The system as defined in claim 1, wherein the processor extracts, from the pre-treatment image, tissue information regarding a biological tissue that is imaged in the pre-treatment image, and estimates the estimated heat diffusion region based on the tissue information and the information regarding the energy supply amount.
 3. The system as defined in claim 2, wherein the processor extracts, from the pre-treatment image, device information regarding the at least one energy device that is imaged in the pre-treatment image, and estimates, based on the device information and the tissue information, the estimated heat diffusion region in a treatment target tissue to be treated by the at least one energy device.
 4. The system as defined in claim 3, wherein the processor estimates regions of the treatment target tissue and the at least one energy device in the pre-treatment image based on the tissue information and the device information, estimates, based on the estimated regions of the treatment target tissue and the at least one energy device, at least one of a tension applied to a biological tissue imaged in the pre-treatment image and a gripping amount of the at least one energy device as an estimation result, and estimates the estimated heat diffusion region based on the estimation result, the tissue information, and the information regarding the energy supply amount.
 5. The system as defined in claim 4, wherein the processor estimates a gripping force of the at least one energy device based on the estimated tension and the pre-treatment image, and estimates the estimated heat diffusion region based on the estimated gripping force.
 6. The system as defined in claim 1, wherein the processor acquires a detection value of a gripping force detected by a gripping force detection sensor provided in a grip section of the at least one energy device, and estimates the estimated heat diffusion region based on the acquired detection value.
 7. The system as defined in claim 1, wherein the processor estimates the estimated heat diffusion region for each temperature or heat, superimposes the estimated heat diffusion region for the temperature or the heat that has been estimated on the pre-treatment image to display the estimated heat diffusion region.
 8. The system as defined in claim 5, wherein the processor estimates a region of a biological tissue and a region of an energy device from the pre-treatment image, and superimposes the estimated heat diffusion region on a region of a biological tissue around the at least one energy device that has been estimated to display the estimated heat diffusion region.
 9. The system as defined in claim 1, wherein the processor estimates the estimated heat diffusion region corresponding to each of plural stages of tension without estimating the tension applied to a biological tissue imaged in the pre-treatment image.
 10. The system as defined in claim 1, wherein the processor estimates the estimated heat diffusion region corresponding to each of plural stages of gripping amount without estimating the gripping amount of the at least one energy device imaged in the pre-treatment image.
 11. The system as defined in claim 1, wherein the processor estimates, based on the pre-treatment image, tension applied to a biological tissue imaged in the pre-treatment image and a gripping amount of the at least one energy device, when a certainty factor in the estimation of the tension is lower than a first predetermined value, the processor estimates the estimated heat diffusion region corresponding to each of plural stages of the tension without using the estimated tension for the estimation of the estimated heat diffusion region, and when a certainty factor in the estimation of the gripping amount is lower than a second predetermined value, the processor estimates the estimated heat diffusion region corresponding to each of plural stages of the gripping amount without using the estimated gripping amount for the estimation of the estimated heat diffusion region.
 12. The system as defined in claim 1, wherein the processor estimates the estimated heat diffusion region corresponding to each of plural stages of gripping force for gripping a biological tissue by the at least one energy device.
 13. The system as defined in claim 1, wherein the processor estimates the estimated heat diffusion region corresponding to each of plural stages of time for the at least one energy device to apply energy.
 14. A computer-readable non-transitory information storage medium storing a program for causing a computer to execute acquiring a pre-treatment image, in which at least one energy device and at least one biological tissue are imaged, in which a state before application of energy from the at least one energy device is imaged, and that is captured by a camera that captures an image of a surgical field, and acquiring information regarding an energy supply amount to the at least one energy device, estimating an estimated heat diffusion region in the pre-treatment image by processing based on a trained model, the estimated heat diffusion region being an estimated range of reach of energy from the at least one energy device after application of the energy based on the energy supply amount, the trained model being trained to output a heat diffusion region from a training device tissue image or a training tissue image, the heat diffusion region being a range of reach of heat from the at least one energy device, the training device tissue image being an image in which the at least one energy device and the at least one biological tissue are imaged, the training tissue image being an image in which the at least one biological tissue is imaged, and superimposing the estimated heat diffusion region on a captured image of the camera and displaying the captured image with the superimposed estimated heat diffusion region on a display.
 15. The information storage medium as defined in claim 14, which stores a program for causing a computer to execute extracting, from the pre-treatment image, tissue information regarding a biological tissue that is imaged in the pre-treatment image, extracting, from the pre-treatment image, device information regarding the at least one energy device that is imaged in the pre-treatment image, and estimating, based on the device information and the tissue information, the estimated heat diffusion region in a treatment target tissue to be treated by the at least one energy device.
 16. The information storage medium as defined in claim 15, which stores a program for causing a computer to execute estimating regions of the treatment target tissue and the at least one energy device in the pre-treatment image based on the tissue information and the device information, estimating, based on the estimated regions of the treatment target tissue and the at least one energy device, at least one of a tension applied to a biological tissue imaged in the pre-treatment image and a gripping amount of the at least one energy device as an estimation result, and estimating the estimated heat diffusion region based on the estimation result, the tissue information, and the information regarding the energy supply amount.
 17. An information processing method, comprising: acquiring a pre-treatment image, in which at least one energy device and at least one biological tissue are imaged, in which a state before application of energy from the at least one energy device is imaged, and that is captured by a camera that captures an image of a surgical field, and acquiring information regarding an energy supply amount to the at least one energy device, estimating an estimated heat diffusion region in the pre-treatment image by processing based on a trained model, the estimated heat diffusion region being an estimated range of reach of energy from the at least one energy device after application of the energy based on the energy supply amount, the trained model being trained to output a heat diffusion region from a training device tissue image or a training tissue image, the heat diffusion region being a range of reach of heat from the at least one energy device, the training device tissue image being an image in which the at least one energy device and the at least one biological tissue are imaged, the training tissue image being an image in which the at least one biological tissue is imaged, and superimposing the estimated heat diffusion region on a captured image of the camera and displaying the captured image with the superimposed estimated heat diffusion region on a display.
 18. The information processing method as defined in claim 17, comprising extracting, from the pre-treatment image, tissue information regarding a biological tissue that is imaged in the pre-treatment image, extracting, from the pre-treatment image, device information regarding the at least one energy device that is imaged in the pre-treatment image, and estimating, based on the device information and the tissue information, the estimated heat diffusion region in a treatment target tissue to be treated by the at least one energy device.
 19. The information processing method as defined in claim 18, comprising estimating regions of the treatment target tissue and the at least one energy device in the pre-treatment image based on the tissue information and the device information, estimating, based on the estimated regions of the treatment target tissue and the at least one energy device, at least one of a tension applied to a biological tissue imaged in the pre-treatment image and a gripping amount of the at least one energy device as an estimation result, and estimating the estimated heat diffusion region based on the estimation result, the tissue information, and the information regarding the energy supply amount. 